Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition

Gesture recognition has attracted considerable attention owing to its great potential in applications. Although the progress been made recently multi-modal learning methods, existing methods still lack effective integration fully explore synergies among spatio-temporal modalities effectively for gesture recognition. The problems are partially due fact that manually designed network architectures have low efficiency joint of multi-modalities. In this paper, we propose first neural architecture search (NAS)-based method RGB-D proposed includes two key components: 1) enhanced temporal representation via 3D Central Difference Convolution (3D-CDC) family, which is able capture rich context aggregating difference information; and 2) optimized backbones multi-sampling-rate branches lateral connections varied modalities. resultant multi-rate provides a new perspective understand relationship between RGB depth their dynamics. Comprehensive experiments performed on three benchmark datasets (IsoGD, NvGesture, EgoGesture), demonstrating state-of-the-art performance both single- multi-modality settings. code available at https://github.com/ZitongYu/3DCDC-NAS .